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(54) Image processing system for dynamic suppression of airbags using multiple model 
likelihoods to infer three dimensional information 



(57) The present invention relates in general to sys- 
tems used to determine whether airbag deployment 
should be disabled or modified due to the proximity of 
the occupant to the airbag. In particular, the present in- 
vention is an image processing system that uses an im- 
age and signal process that utilizes real-time streaming 
video-images from a video camera to determine if the 
occupant is too close to the air bag, or will be to close 
the air bag by the time that the airbag deploys . I n a crash 
situation, the system quickly detennines whether or not 
the airbag should be disabled or modified. The process 
uses a multiple-model Kalman filterto infer three-dimen- 
sional information from a sequence of two-dimensional 
images. The system predicts the position and shape of 
the occupant at a faster rate than the rate in which the 
camera collects data. 
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Description 
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^'^T", J^'' 3^"^'^' ^y^*^'"^ "^^'^ ^° determine whether the deployment of an alrbaa 

Should be prevented or modified due to the proximity of the occupant to the airbag. In particular the orient tnvenSon 
.s an .mage processing system that receives a stream of two-dimensional images and Jp^tes SeraXr^rprSi^^^ 
weighted processes to infer three-dimensional characteristics to those images and preSS whelS the uJSo^^^^^^ 
he^xupant would be within the At-Rlsk-Zone ("ARZ") by the time that an air bag would ISaiydw' 
[00021 Conventional a.rt,ag deployment systems have contributed significantly to the safety of Sams in automo 

Ivste^XmlM '^'^^^ <^'sablement systems are knownTn the a^ Sor^e of'th^se 

pSS^s wrtCr^r^aT ""^"'^^ "^"^^^ ''^ ^^^^^ • - --al slgn^^nt 

Khir "'■^^■/"^^'.fy^te"^^ require highly expensive cameras. Timing is critical to any system used to detemiine 
whether or not an a.rbag should be deploy A standard video camera operates at a frequent Syeen 50 I^Snd 

quentupdates.of approximately 200updatespersecond(200hz). iVloreover.it wouldbed 

Z^TLTk , "f ""^ ^'^'^'^ '"^^^^'^ of '"^r^'y '-^entifying the occupanTsTocSonTthe 

tirne that the image is captured, it would also be desirable if accurate predictions coi^d be generated at al^tefrlte 

S's^^dT^^^^^^^^^^ 

?^}irnlZ°"'^'f'°'^'',t^^'^^ ^^^^'^ susceptible to "noise- because prior art systems focus solely on the most 
recent .mage and ignore the senes of images captured mere fractions of a second earlier "Noise" resulte from seS 

f«: ors.,ndud,ng the inherent imperfections of the segmentation process Which is theprocossoTe«^^^^^ 
S^uolm 'rH'T''" °' ,r"P«"*- '^o'^""" fr<"" *e surrounding area, from the ambient imagjln image Tf the 
occupant and an .mage of the surrounding area. It would be desirable for an image processing system to utNize an 
woX r^tZTn' !""'«9^*« the infomiatlon contained in the most recent image into a coSenslve S^me 

be 3Llir«hL ^"'^ '"•''^^'^'y' ^^^^ t° "^^X^ prior predictbns It wouSso 

to inSfn ' r^'^f^"* to be weighted predictions. IVIoreover. It would be helpful for such weighted preSStons 

away froX d^^^^^^ ""'t ^'^^^^ ^ '^^"'"9 '^^^^ drLr. leaning St 

?M051^i^hl H n ■ h ^'«'"l"P".9''t. and predefined occupant modes such as crash, stationary, or human. 

bv lLr« il . 7 '^^y ^^ly two-dimensional images, fhe Images captured 

SoZ^n ' ,^ J"^- ^r^'^"' '"'^'^"^'y ^° 'li'^eisional images. It would be useful if three d^m^loS 
^ZtTJt '"'^"^ ^ ^"^ <l""e"«'onal Images. Moreover. It would be help ufif S.ned 

^f^r^^SL^tZgr""^' '^^'^^^^ °' « three-dimensiona, intom,alIon f^ltiS 

SUMMARY OF THE INVENTION 

S™t t!^'^ '^L^'^^ '° ^" ""^^^ processing system used to detemiine whetherthe proximity of an occupant 

Ih^^Z ' '""^^'Tl' ^" '"^^^^ processing system that incorporates current measuremente wSh past 

pred^tions in order to detemnine the location of an occupant relative to the position of an airbagXrymTnt system 
£o 2 „I ' T"""' "^^'^ ""^'^ t° incorporate a measurement of the m^recem ima^^ 

Si 7c3r^rslrir''''''°"' """^ '"«««"'-ements relating to the position and shape of the'^upant 
I0008I A camera or some other sensor is used to capture an ambient Image, the image of the occuoant and the ar^l 

tU^ZtT^'V^'T "^^^^ occupan?with the s'urrrding arrimoJ^^^^ 

fMno nrfc^rr; r? ' r^^"" *^g'"«"t««°n- The segmented image is then subjected to an shZ 

hS d«wnTtH ^" """^^ "'^"^ *° "PP^^ ^^'■"o °' «^"P«"'- A lower shape is fitted f om the o^upaSs 
Shfr, °~"P^"tVeet- The two Shapes can overlap at one point, the centr<^d of the occupant InThe a^ea 

c'lrirtre^Verellipsr ''^'^^ ^"'"^^ ^ --"t cJml^ris"^' 

S^'^I^iI?""!?^-^"'!^'^'*^'"^ processes are perfom«d using multiple model Kalman filters. The process involves 
two parallel and s.multaneous sub-processes. One process tracks important infomiation relating to thTsl^De^f the 

SctlS . J? f.^^P^.'^^'^*'- P^««oto^ process tracks and predicts the shape of the occupant by tracking and ore- 
dicting (collecfvely "shape variables"): the "major axis" of the ellipse, a bisecting line through'the ion^ri slion 
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of the ellipse; the "minor axis" of the ellipse, a bisecting iine through the shorter cross section of the ellipse; and the 
y-coordinate or "height" of the centrold In the segmented image. For each shape variable, the velocity and acceleration, 
in addition to position, are each tracked and predicted. 

[0011] An occupanfs shape is defined into one of several predefined "states" such as leaning towards the driver 
5 ("left'), leaning away from the driver ("right"), and sitting upright ("center"). All processing by the shape tracker is mod- 
eled in each of the predefined states. Using the probabilities associated with potential transitions from one state to 
another state, and incorporating past predictions through the use of an iterative and weighted Kalman filter, the shape 
tracker and predictor will detennine which state the occupant is In by detemriintng which model ellipse most closely 
resembles the most recently measured ellipse. Thus, even though the tilt angle towards or away from the driver (the 
10 "sideways tilt angle" O), is perpendicular to the image plane and thus is not visible In a two-dimensional image, the 
system can Infer valuable three-dimensional infomriation about a two-dimensional image. 

[0012] The detemal nation of shape state is then used to update the estimate for all shape variables. A shape estimate 
is updated using incoming measurements, state predictions, and the Kalman filter gain matrices. The shape tracker 
and predictor can be used to generate estimates more frequently than a camera can capture images. Thus, the system 

15 does not require the use of a high-speed video camera. 

[0013] The motion tracker and predictor processes infomriation relating to the position, velocity, and acceleration of 
the occupant. "Motion variables" include the x-coordinate of the centrold which is used to measure the "distance" 
between the x-coordinate and the airbag and the "forward title angle" 8, the degree to which the occupant is tilted 
towards the front of the vehicle. 

20 [0014] Position, velocity, and acceleration are tracked and predicted for all motion variables. An occupant's motion 
is defined into one of several predefined "modes" such as "stationary," "human," or "crash". AH processing by the motion 
tracker and predictor is modeled in each of the predefined modes. Predictions are generated for each mode. Using 
the probabilities associated with potential transitions from one mode to another mode, and incorporating past predic- 
tions thought the use of an iterative and weighted Kalman filter, the motion tracker and predictor will generate one 

25 combined estimate for each motion variable. The motion tracker and predictor can be used to generate estimates more 
frequently than a camera can capture Images. Thus, the system does not require the use of a high-speed video camera. 
[0015] The final step Is the actual determination of whether or not the occupant Is within an At-Risk-Zone ("ARZ"). 
An ARZ Intrusion Indicator determines whether the upper torso of the occupant would be within the ARZ at the time 
in which an airbag would deploy. The ARZ Intrusion Indicator also detennines whether the person Is leaning so far 

30 towards the driver that they miss the ARZ. For different vehicle models, the ARZ will be set differently. This calculation 
is updated very quickly using the outputted state transition matrix which allows the system to predict ahead the position 
and shape at a rate much more quickly than the rate at which the camera collects data. 

[0016] Various aspects of this invention will become apparent to those skilled in the art from the following detailed 
description of the preferred embodiment, when read in light of the accompanying drawings. 

35 

BRIEF DESCRIPTION OF THE DRAWINGS 



[0017] Fig. 1 shows a partial view of the surrounding environment for one potential embodiment of the invention. 

[0018] Fig. 2 shows a high-level process flow of the use of the image processing system. 

40 [0019] Fig. 3 shows a high level block diagram of an image processing system. 

[0020] Fig 4 shows the application of the ellipse fitting process. 

[0021] Fig. 5 show an upper torso ellipse with all corresponding shape variables and motion variables (excluding ^ 
velocity and acceleration derivations). 

[0022] Fig. 6 shows three potential shape states and three potential sideways tilt angles. 

^5 [0023] Fig. 7 shows a markov chain of probabilities relating to the three shape states of left, right, and center. 

[0024] Fig. 8 shows a markov chain of probabilities relating to the three motion modes of human, stationary, and crash. 

[0025] Fig. 9 shows the detailed process flow for detemnining the shape of the occupant. 

[0026] Fig. 1 0 shows the detailed process flow for determining the motion of the occupant. 

50 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

A. PARTIAL VIEW OF SURROUNDING ENVIRONMENT 



[0027] Referring now to the drawings, illustrated in Fig. 1 is a partial view of the surrounding environment for one 
potential embodiment of the invention., an image processing system 16. If an occupant 1 8 Is present, the occupant 18 
sits on a seat 20. In the preferred embodiment, a camera or other sensor 22 is attached In a roof liner 24, above the 
occupant 1 8 and closer to a front windshield 26 than the occupant 1 8. The camera 22 should be placed in a slightly 
downward angle towards the occupant 1 8 in order to capture changes In the angle of the occupant's 1 8 upper torso 
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through ,ong wires. An airt,ag controller 32 KhTwn i ll^^ f"-- "Pa- 

stil! function even if the airbag deployment svstem w^rp lorltfH in = h « ' P'^^^nt Invention could 



B. HIGH LEVEL PROCESS FLOW 



C. COMPUTER SYSTEM PROCESS FLOW 

is performed which extracts aTeglnted imaqe 42 of th J oll^f """""'^^ "l"'^"" ^^'"^"t^"''" «>"tine 40 
not the surrounding seat area 2rf rom the Imbfent I^^Ia'^^ ' '"^^ '"'^'"''^ '^"'y °«="P«"t 
existing art. Segmentation 40 1 accompli hed^^^^^^^^^^^ ^'"^^^ ^ k"ov.n under 

be performed in a computer separate f^m ^eXu"^^^ 

^d^o:;d^reTp;rr:st;rcr^^^^^^^ rr"^^^ ""'"^ -^^--^p- 

down to the occupant's hips A IweSse s fL^fro^tr everything from the occupant's 18 head 

ellipses overlap at one pontwhlTinSlr^^^^^ *° ''^^ ocoup^nVs 18 hips. Both 

center of the upper elli£e 58 The ^ToSt 0?!^^ r « "^""''"'^ P'-^'^'-^^'V as the 

w«hthedlscusSronrtlrg1oF^s4 5a^ 

Shape tracker 48 tracl« variabtesrSna t? r"^«^^^ sub-processes simultaneously with each other. A 

below in the discussion ZtS^^ t^pTq I A 1^^^^ ^"^'^ """"^^ ^^^'=^"'^<^ Sweater detail 

ellipse, speclffcaliy w«h re^'Xel'rizonUM'^^^^^^^^^^ "^'7' 
36 In the instrument panel 34 "mis process is dP-snnhlH ^ , ! °«=upant 18 and the airbag deployment system 
[0032, Every embodimenfof^lJ^T^ersyrte^^^^^^^^ 

the position, velocity and acceleration nf r-^^I-^l tracking and predicting system 46 track and predict 

of the occupan T8 uZ a mullolfllTrTt '^'^""9 ^^^^ upper torso 

series of on^oino ^ 'Z:™^^^^^^ '"corporates the latest measurement into a 

dieting system ^ mcoioraTes whatl'kn^^^^^^ tracking and pre- 

it incorporates past predictions anS r^a^uZe JiZw^^^^ "''^ ^'^^ P°^^^"' 

even future states, and it can do so eJen whTn thl nrt7«! „ f , i ' """'^^ estimations of past, present, and 
paper entitled "An ntroductten to the KaJnaS Fmer T^J^T^ the modeled system Is unknown. An academic 

reference. -n,e general e^uattoM^theTaLX^^^^ ^^'^ ^"^"^^ ^^'PO.Bted by 

X(newpredle«on) = ^(old predWion) + Gain[-X(„y p^,^^, + X(„^„^J 

estimate X^^w estimate) esiimate X(^^ estimate) IS simply the value of the old 
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(new estimate) ~ ^(old estimate} ^['^(old estimate) ^(measured)] 



(new estimate) - ^(old estimate) 



^(new estimate) " '^(oW esUmate) 

10 A Gain of 1 indicates such confidence in the most recent nneasurement X^^easured) ^^^^ prediction X^^^ew estimate) 

is simply the value of the most recent measurement X^measured). 

^(new estimate) = ^(old estimate) ^ ["^(oid estimate) **• ^(measured)! 

15 

^(new estimate) ~ ^(old estimate) *^(old estimate) ^(measured)l 

20 ^(new estimate) = X^^gggy^ed) 

In a real world application, the Gain is virtually always greater than 0 and less than 1. The Gain thus detemriines to 
what degree a new measurement can change the previous aggregate estimate or prediction of the location of an object, 
in the case of the instant invention, the occupant 18 is the object being tracl^ed. Both the shape tracker 48 and the 

25 motion tracker 50 are described in greater detail below, along with Figs. 9 and 1 0 respectively. 

[0033] The output from both the shape tracker 48 and the motion tracker 50 are sent to an At-Risk-Zone ("ARZ') 
Intrusion Indicator 52. The ARZ Intrusion Indicator 52 determines the proximity of the closest point of the upper ellipse 
relative to the instrument panel 34, the distance which defines the ARZ. The ARZ Intrusion Indicator 52 also detemnines 
whether or not the occupant 18 is leaning so far toward the driver that the occupant 1 8 will miss the ARZ altogether. 

30 Calculations in the ARZ Intrusion Indicator 52 are done more quickly than the rate in which the camera 22 collects 
data. The ARZ Intrusion Indicator 52 is specifically configured for the particular model of vehicle. Thus, a minivan will 
typically have a different ARZ than a compact automobile. The ARZ Intrusion Indicator is discussed in greater detail 
below, along with the discussion relating to Fig. 5. 

[0034] The detemiination by the ARZ Intrusion Indicator 52 is sent to the airbag controller 32 so that the airbag 
35 controller 32 can implement the appropriate response with the airbag deployment system 36. The deployment of the 
airbag at 36 can either be totally disabled or merely modified as a result of the ARZ determination made at 52. 

D. ELLIPSE FITTING ROUTINE 

40 [0035] Fig. 4 illustrates the ellipse fitting routine 44 implemented by the computer system 30. The upper ellipse 58 
extends from the hips up to the head of the occupant 18. The lower ellipse 56. extends down from the hips to include 
the feet of the occupant 1 8. If the entire area from an occupant's 1 8 hips down to the occupant's 1 8 feet is not visible, 
a lower ellipse is generated to represent what is visible. Both ellipses overlap at a point that can be known as the 
centroid 64, although the centroid 54 is preferably the center of the upper ellipse 58. In non-preferred embodiments, 

45 shapes other than ellipses are used to represent the upper and lower parts of an occupant 1 8. The ellipse fitting routine 
is well known in the art. The image processing system need not utilize the lower ellipse 56, and it is preferable to 
generate the centroid 54 with respect to the upper ellipse 58. 

[0036] Fig. 5 illustrates many of the characteristics that can be inputted into the computer system 30 from the ellipse 
fitting routine 44. Motion characteristk^ include the x-coordinate ("distance") 60 of the centroid 54 and the forward tilt 

50 angle ("6") 62. Shape measurements include the y-coordinate ("height") 58 of the centroid 54, the length of the major 
axis of the ellipse ("major") 64 and the length of the minor axis of the ellipse ("minor") 66. Rate of change information, 
such as velocity and acceleration, are also captured for all shape and motion measurements, so in the preferred em- 
bodiment of the invention there are nine shape characteristics (height, height', height", major, major*, major", minor, 
minor*, and minor") and six motion characteristics (distance, distance', distance", 6, 6', and 6"). The sideways tilt angle 

55 o is not shown because it is perpendicular to the image plane,- and this the sideways title angle 0 is derived, not 
measured, as discussed in greater detail below. 

[0037] Fig. 6 illustrates the sideways tilt angle "(^") 73. In the preferred embodiment of the invention, there are only 
three shape states, leaning left towards the driver (left) 76, sitting upright (center) 72, and leaning right away from the 
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E. MARKOV PROBABILtTY CHAINS 



0 



5 



equation for each shape state. Similar ea^hrSon i^w . ? ^ separate Kalman filter 

motion mode, in every embo«mZ o,The ?veZTtt^^^^ ^ "^"^'^'^ ""^^ ^'I^^""" ^^^h 

motion mode. There are ceiZ^pMi^ZlSt^LT ''T. 1^' ^* ^^"^^ one 
J.esepro.a.l.escant,est^^E^^^ 

s.m:nta^r:pr^^^^^^^ 

from the driver CYlghf) 82 The probabXTan o«^^,nf„t k ^ ' °' ^'"'"^ "P"^''* 78, or leaning away 

state can be identmed by lines oTg^at S a rpaSr shaof It ''^T'"' ^"^ ^""^'"^ ^ 
shape state. For example the probablSl of 1 o™nf ? ® "^'^ P"""*'"9 If^e subsequent 

by the arrow at 84. T^Zob^m^ Z^m^^^^ T^!"^'^ ^^'""'"'"S "^"'^^ ^'^'^ -^"^ ^-P^esented 

o;pngfromoe.ertoVpc%.srS3^S^^^^^ 



rSirThe ZT^ P-'>«'"''«-^ -'ginating from any particular state must ateo add up to 1 0 

zz of :L^.^:lrify^;reTrr:t"s csStrp'^tjr.rrr;^ ^"^^-•"-'^ 

next interval of time, and the arro^^t 90 r^^se^the D^S^St^ ^ '^""^"^ ^'^ht P--" by the 

.eft P-. The sum Of all possible proba^irrC.r,rr^^^^^^^ 



pL-C + p'-'-^.pL-R^,_0 



P-'L S ^ 88T;retl^;TSbi.l^ ^ "^'"^ to the right 

arrowat94represents,hep«^tranS^^^^^ 

originating from an initial tilt state of right equals 1 '«« P" ■■• The sum of all possible probabilities 

p"-'= + p"-'- + p"-" = i.o 

Sr^Le'^eaK^^rS i^^^^^^^^ St^f f^^^'^/^rn r^"'' embodiment captures between 50to 
occupant, or for a right 82 leaning oc^rpan ^bZl «o f ° '° "^''^^ « 82 leaning 

more likely that a left 80 leaning oc^u^rw , "1^?!°^ !!! , ! , « ""^^^ ^'S" « «««»"<^- ^ far 

and similarly it is far more realEor a Lft flJ ?! ''^^^r state 78 before becoming a right 82 leaning occupant 

a right 82 leaning occZn tS in he p^er S Z^r? ? ^ '^"'"^"'^ ''^ '^"P^"' 

a.so always be/etat.L.Th;Srr.Xi!^S"rr:g:^^^^^ 



pR<'.^p''-R=l.o 



10043] Fig. 8 il,ust«tes a similar MarKov ch«n to represent the relevant probabilities relating to motion modes. The 
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preferred embodiment of the invention uses three motion modes: stationary 1 02, represents a human occupant 1 8 in 
a mode of stillness, such as while asleep; human 1 04, represents a occupant 18 behaving as a typical passenger in 
an automobile or other vehicle, one that is moving as a matter of course, but not in an extreme way; and crash 122, 
represents the occupant 1 8 of a vehicle that is in a mode of crashing, 

[0044] The probability of an occupant being in a particular state and then ending in a particular state can be identified 
by lines originating in the current state with arrows pointing to the new state. For example, the probability of an occupant 
in a stationary state remaining In stationary state P^-^ is represented by the arrow at 1 08. The probability of moving 
from stationary to human P^-h is represented by the arrow 114 and the probability of moving from stationary to crash 
P^'C is 90. The total probabilities resulting from an initial state of stationary 102 must add up to 1 . 

pS-S^pS.H_^pS.C^^_Q 



[0045] Similarly, the probability of human to human is P^-^ at 116, human to stationary is P^-^ at 112, and human 
IS to crash is P*^'^ at 120, and stationary to crash P*^*^ is 90. The total probabilities resulting from an initial state of 
stationary 102 must add up to 1 . 



pH-H _^pH-C ^ pH-S ^ ^ 



[0046] Lastly, the probability of going from crash to crash is P^-^ at 1 24, crash to stationary is P^-s at 1 1 0, and crash 
to human is P^" at 118. The total probabilities resulting from an initial state of crash 122 must add up to 1 . 



pOC + p^S^pC-" = 1.0 



[0047] As a practical matter, it is highly unlilcely but not impossible for an occupant 1 8 to ever leave the state of crash 
at 122 once that state has been entered. Undermost scenarios, a crash at 122 ends the trip for the occupant 18. Thus, 
in the preferred embodiment, P^-^ is set to a number very close to zero and P^s js also set to number close to zero. 
30 The three equations for motion mode probabilities in the prefen-ed embodiment are as follows: 

pC-C ^ p-C-H _^ pC-S ^^^Q 



Vp"-^+p"-® =1.0 



pS^_^p&H_^pS.S ^^^Q 



[0048] The transition probabilities associated with the various shape states and motion modes are used to generate 
a Kalman fitter equation for each combination of characteristic and state. The results of those fitters can then be ag- 
gregated in to one result, using the various probabilities to give the appropriate weight to each Kalman filter. All of the 
probabilities are predefined by the user of the invention. 

[0049] The markov chain probabilities provide a means to weigh the various Kalman filters for each characteristic 
and for each state and each mode. The tracking and predicting system 46 incorporates the markov chain probabilities 
in the fomn of two subsystems, the shape tracker and predictor 48 and the motion tracker and predictor 50. 

F. SHAPE TRACKER AND PREDICTOR 



[0050] Fig. 9 discloses a detailed flow chart for the shape tracker and predictor 48. In the preferred embodiment of 
the invention, the shape tracker and predictor 48 tracks and predicts the major axis of the upper ellipse ("major"), the 
minor axis of the upper ellipse ("minor"), and the y-coordinate of the centroid ("height"). Each characteristic has a vector 
describing position, velocity, and acceleration information for the particular characteristic. The major vector is [major, 
55 major*, major"], with major* representing the rate of change in the major or velocity and major" representing the rate of 
change in major velocity or acceleration. Accordingly, the minor vector is [minor, minor*, minor*], and the height vector 
is [height, height*, height"]. Any other shape vectors will similariy have position, velocity, and acceleration components. 
The first step in the shape tracking and prediction process is an update of the shape predtotion at 1i26. 
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1. Update Shape Prediction 

[0051] An update shape prediction process Is performed at 126 This orocess talcp<. th^ lact eho« * 
extrapolates that estimate into a future prediction using a transition nJairix "'^^^ 

Updated Vector Prediction = Trans/ffon /M!aWx 'Lasr VfectorEstf/nafe 

t'^he'c^^^^^^^^^ 

atiaoas^escjedbe"^^^^^^^^ 
The pr< 
lnvo)(ec 
[0052] 



Lne occupant 1 8 will be on the basis of its past position, velocity and accelerat nn Tho loo "7 \ 7 
at 130 as described below. The process from 126 tn \7r f^^ . o^^^ ^^^'"^^^6 is produced 

(1 At !4At^) (X ) 



Updated Vector Prediction => (0 1 At ) • (x 



) 



(0 0 1 ) (x» ) 



[0053J The updated shape vector predictions are: 

Updated major for center state. 
iJpdated major for right state. 
Updated major for left state. 
Updated minor for center state. 
Updated minor for right state. 
Updated minor for left state. 
Updated height for center state. 
Updated height for right state. 
Updated height for left state. 



2. Update Covariance and Gain IVIatrices 

covarianU a<SS toJe^^ In the piltlm^^^^^^ "P'*^*^*^ ' ^S. The shape prediction 

most recent measuremltTstor^^^^^^^ 

estirnate covariance accounts for error ?n me^str;L;^ 

a^l^l'Z'Z:^^'"^"''' ""'^'^^ "^^^ Shape precision covari- 



Siiape Prediction Covariance Matrix = 
[State Transition Matrix * Oid Estimate Covariance Matrix ' 
transpose(State Transition Matrix)] + System Noise 
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noise in the system. The constants used in the system noise matrix are set by the user of the invention, but the practice 
of selecting noise constants are Icnown in the art. 

[0056] The next matrix to be updated is the gain matrix. As discussed above, the gain represents the confidence of 
weight that a new measurement should be given. A gain of one Indicates the most accurate of measurements, where 
5 past estimates may be ignored. A gain of zero Indicates the least accurate of measurements, where the most recent 
measurement is to be ignored and the user of the invention is to rely solely on the past estimate instead. The role 
played by gain is evidenced in the basic Kalman filter equation where 

iO ^(new estimate) = ^(old estimate) Gain[-XjQ|j estimate) ^(measured)] 

[0057] The gain is not simply one number because one gain exists for each combination of shape variable and shape 
state. The general equation for updating the gain is as follows: 

IS 

Gain = 

Shape Prediction Covariance Matrix * transpose(Measure Matrix) * 
^ inv(Residue Covariance) 

The shape covariance matrix Is calculated above. The measure matrix is simply a way of isolating and extracting the 
position component of a shape vector while Ignoring the velocity and acceleration components for the purposes of 
determining the gain. The transpose of the measure matrix is simply [1 0 0]. The reason for isolating the position 

25 component of a shape variable is because velocity and acceleration are actually derived components, only position 
can be measured by a snapshot. Gain is concerned with the weight that should be attributed to the actual measurement. 
[0058] In the general representation of a Kalman filter, X(„^^ estimate) = X(oid estimate) + Ga'n[-X(oid estimate) + X(measured)]. 
the residue represents the difference between the old estimate and the new measurement. There are entire matrices 
of residue covariances. The inverse of the residue covariance matrix is used to update the gain matrix. It is known in 

30 the art how to tal<e the inverse of a matrix, which is a simple linear algebra process. The equation for residue covariance 
matrix is as follows: 

Residue Covariance = 

35 . 
[Measurement Matrix * PredSction Covariance * transpose(Measurement Matrix)] + 

Measurement Noise 

40 The measurement matrix is a simple matrix used to isolate the position component of a shape vector from the velocity 
and acceleration components. The prediction covariance is calculated above. The transpose of the measurement 
matrix is simply a one row matrix of [1 0 0] instead of a one column matrix with the same values. Measurement noise 
is a constant used to incorporate error associated with the sensor 22 and the segmentation process 40. 
[0059] The last matrix to be updated is the shape estimate covariance matrix, which represents estimation error. As 

45 estimations are based on current measurements and past predictions, the estimate error will generally be less sub- 
stantial than prediction error. The equation for updating the shape estimation covariance matrix is as follows: 

Shape Estimate Covariance Matrix = 

50 

(identity Matrix - Gain Matrix * Measurement Matrix) * 
Shape Predictor Covariance Matrix 

55 [0060] An identity matrix Is known in the art, and consists merely of a diagonal line of 1 's going from top left to bottom 
right, with zeros at every other location. The gain matrix is computed and described above. The measure matrix is also 
described above, and is used to isolate the position component of a shape vector from the velocity and acceleration 
components. The predictor covariance matrix is also computed and described above. 
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3. Update Shape Estimate 

[00611 An update shape estimate process is invoked at 138. The first step in this process is to compute the residue. 

Residue = Measurement - (Measurement Matrix * Prediction Covariance) 
Then the shape states themselves are updated. 

Updated Shape Vector Estimate = Shape Vector Prediction HGain * Residue) 
When broken down into individual equations, the results are as foliows: 

Xfmajoratt) = X(majoratt) + Ga«"[-Xfmajor at t-1)+ ^teasuf^ m^,)] 
X(maJoratt) = x}-major at t)+ ^^^^['^\n^a\oratU^) + ^ {"measured ma|o,)] 
X(maior at t) = X(n«Jor at f)+ Gm-X^„^, 3^ + xf^^ ^^^^^J 
Xfmlnor at t) = ^fnAror at t)+ Ga'n[-XS.Inor at t-l) + X<^^^ 
Xl'mmoratl) = X('mlnoratt)+ Gain[-X}-„,„, ^ )+ X{-^„^„^„^] 
X(minoratt) = X(„,noratt)+ ^^^^^^l'^ (minor at^.^) + X("„,ea8ured minor) 
^(helghtatt) = X(height at t)+ Ga^"t-Xj^eight at 1-1)+ ^fmeasured height)! 

35 

xUght at t) = Xj-heigh, at t)+ (^^H->i'^^ at M) + x{-^3„„^ 

R R p 

40 X(he,gh, at t) = X(h,igh, a. t)+ Gainf-Xj^^^^ ^ + X^„^^^ 

In the preferred embodiment. C represents the state of center. L represents the state of leaning left towards the driver 
and R represents the state of leaning right away from the driver. ' 

45 4. Qenerate Combined Shape Estimate 

iteD ai 13?^e fiS^IItV?^^""^ "'^'^ ^"'^ is a generate combined shape estimate 

IS recalculated, using the same fomiula as discussed above. covariance 

Covariance Residue Matrix = 
^Measurement Matrix * PreOcOon Covariance Matrix * 
tianspose(Measurement Matrix)] + Measurement Noise 
[0063] Next, the actual likelihood for each shape vector Is calculated. The system 16 detemiines which state the 
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occupant is In by comparing tlie predicted values for the various states witli the recent best estimate of what the current 
values for the shape variables actually are. 

LikeUhood( R ) = e"^"*^-"*** 
( L ) 

There is no offset in the preferred embodiment of the invention because it is assumed that offsets cancei each other 
10 out, and that the system's 1 6 processes. Sigma represents variance, and is defined in the implementation phase of 

the invention by a human developer. It is Icnown in the art how to assign a useful value for sIgma by looking at data. 

[0064] The state with the highest likelihood determines the sideways tilt angle O. If the occupant 1 8 is in a centered 

state, the sideways tilt angle is 0 degrees. If the occupant 1 8 is tilting left, then the sideways tilt angle is - O. If the 

occupant 18 is tilting towards the right, the sideways tilt angle is O. in the preferred embodiment of the invention, O 
15 and • O are predefined on the basis of the type and model of vehicle using the system 1 6. 

[0065] Next, state probabilities are updated from the likelihood generated above and the pre-defined markovian 

mode probabilities discussed above.. 
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pC^pOC^pR^^pL-C 



pR _ pR-R ^ pC-R 



The equations for the updated mode probabilities are as follows, where L represents the likelihood of a partk^ular mode 
as calculated above: 

Probability of mode Left = 

Probability of mode Right = 
I/IL"- * (P*-"*- + P^*-) +l" * (P""" + P^"")+L^ *(P^^ + P"^ + P*--^)]* l" '(P^^+P^") 

Probability of mode Center = 

^^[\}■ *{p'-'- h-p^Vl" -(p^'^+p^'" • (p^^ + p"'^ + p*-"^)]* *(p^^+p*-'^+p^"^) 

[0066] The combined shape estimate is ultimately calculated by using each of the above probabilities, in conjunction 
with the various shape vector estimates. 

so X = ProbabiUty of mode Left * 

+ Probability of mode Right* X'"^*'* 
+ Probability of mode Center* x*^"*" 

55 X is any of the shape variables, including a velocity or acceleration derivation of a measure value. 
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Q. MOTION TRACKER AND PREDICTOR 

^o- ""^J^l^^^" predictor 50 in Fig. 1 0 functions similarly in many respects, to the shape tracker and 
predictor 48 .n Frg. 9. The motion tracker and predictor 50 tracks different characteristics and veSorslEan^he sh^f 
tracker. In the preferred embodiment of the invention, the x-coordinate of the centroid ani ^eflr^^^SZ^eB^^t 
and their corresponding velocities and accelerations (collectively "motion variables") are traoZ^nd pSed The 
x-coord.nate of the centroid is used to determine the distance between the occupant 18 and a tocaSrSn^Ie 

^.n^fJ^TT^^. "^"^ '"""''^ ^ P"""'"" component (x). a velocity component (x'), and an SeraZ 
component (9 ). Any other motion vector will similarly have position, velocity, and acceleration componente 
1. Update Motion Prediction 

[0069] An update motion predfction process is performed at 134. This process takes the last motion estimate anri 
extrapolates that estimate into a future prediction using a transition matrix ^ ^"'^ 

Updated Vector Predlctton = Transition Matrix * Last Vector Estimate 

at l3r«rH.V.ririH K r °' "'P^^' ^^•"'^^y' ^"•^ acceleration. The last vector estimate Is produced 

at 138 as descnbed below. The process from 134 to 136. from 136 to 138. and from 138 to 140 looDS b^ to A? 

Z^TT^rV^ ^" ^'''""^'^ P^^^'-^^IV generated at 38, so procilTng ^d^e is not 

invoked the first time through the repeating loop that is steps 134-140 at 1 34 and 136 is not 

[0070] The following equation is then applied for all motion variables and for all motion modes- 



(1 At YiA^) (x ) 

Updated Vector Prediction = (0 1 At ) • (x' ) 

(0 0 1 ) (x" ) 

Ihrl^S^r^^ 7"°*:"^' '^"^'^ "Pdated vector predictions at 134 because there are 

Sns aS: """"""^ «-bodiment. and 3 x 2 = 6. The updated moJoTp'" 

Updated x-coordinate for crash mode. 
Updated x-coordinate for human mode. 
Updated x-coordinate for stationary mode. 
Updated e for crash mode. 
Updated 9 for human mode. 
Updated 6 for stationary mode. 

2. Update Covarlance and Gain Matrices 

SilrL "P^«t^^ a" 'potion variables and all modes at 134, the motion prediction 

motrrrt • ^^'"^^'^ ^^trices must be upd^J^at lS 

rnotron prediction covanance accounts for error in the prediction process. The gain, as described above 
the weight that the most recent measurement is to receive and accounts for errors In the me^^^^^ 

""'^ "P^^*- -tlon pred^lon covar- 



Motion Prediction Covanance Matrix = 
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State Transition Matrix * Otd Estimate Covariance /Matrix * 
transpose(State Transition Matrix) + System Noise 

5 

The state transrtion matrix is the matrix that embodies Newtonian mechanics used above to update the motion predic- 
tion. The old estimate covariance matrix is generated from the previous loop at 136. On the first loop from 1 34 through 
140, step 134 Is skipped. Taking the transpose of a matrix Is simply the switching of rows with columns and columns 
with rows, and is known under the art. Thus, the transpose of the state transition matrix Is the state transition matrix 
10 with the rows as columns and the columns as rows. System noise is a matrix of constants used to Incorporate the idea 
of noise in the system. The constants used In the system noise matrix are set by the user of the invention, but the 
practice of selecting such constants are known In the art. 

[0073] The next matrix to be updated is the gain matrix. As discussed above, the gain represents the confidence of 
weight that a new measurement should be given. A gain of one indicates the most accurate of measurements, where 
IS past estimates may be ignored. A gain of zero indicates the least accurate of measurements, where the most recent 
measurement is to be ignored and the user of the invention is to rely on the past estimate instead. The role played by 
gain is evidenced in the bask: Kalman filter equation where 

20 ^(n«w estimate) = ^(old estimate) ^®'"["^(old estimate) + ^(measufod)l 

[0074] The gain is not simply one number but an entire matrix because one gain exists for each combination of 
motion variable and motion mode. The general equation for updating the gain is as follows: 

25 

Gain = 

Motion Prediction Covariance Matrix * transpose(Measure Matrix) * 
inv(Residue Covariarjce) 

The motion covariance matrix is calculated above. The measure matrix is simpty a way of isolating and extracting the 
position component of a motion vector while ignoring the velocity and acceleration components for the purposes of 
determining the gain. The transpose of the measure matrix is simply [1 0 0]. The reason for isolating the position 
35 component of a motion variable is because velocity and acceleration are actually derived components. Position is the 
only component actually measured, and because gain is concerned with the weight that should be attributed to the' 
actual measurement, derived variables should be isolated. 

[0075] In the general representation of a Kalman filter. X^^^^ estimate) = ^(o\d estimate) + Gain[.X(o,d estimate) ^ X(measured)]. 
the residue represents the difference between the old estimate and the new measurement. There are entire matrices 
40 of residue covariances. The Inverse of the residue covariance matrix is used to update the gain matrix. It Is known in 
the art how to take the inverse of a matrix, which is a simple linear algebra process. The equation for residue covariance 
matrix is as follows: 

^ Residue Covariance = 

[Measurement Matrix * Prediction Covariance * 
transpose(Measurement Matrix)] Measurement Noise 

50 

The measurement matrix Is a simple matrix used to isolate the position component of a motion vector from the velocity 
and acceleration components. The prediction covariance Is calculated above. The transpose of the measurement 
matrix is simply a one row matrix of [1 0 0] instead of a one column matrix with the same values. Measurement noise 
Is a constant used to incorporate error associated with the sensor 22 and the segmentation process 40. 
55 [0076] The last matrix to be updated is the motion estimate covariance matrix, which represents estimation error. As 
estimations are based on current measurements and past predictions, the estimate error will generally be less sub- 
stantial than the prediction error. The equation for updating the motion estimation covariance matrix is as follows: 
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Motion Estimate Covariance Matrix = 
(identity Matrix - Gain Matrix * Measurement Matrix) * 
Motion Predictor Covariance Matrix 

3. Update Motion Estimate 

'5 [0078J An update motion estimate process is invoked at 1 38. The first step In this process is to compute the residue. 

Residue = Measurement - (Measurement Matrix * Prediction Covariance) 
20 Then the motion states themselves are updated. 

Motion Vector Estimate ^Motion Vector Prediction + (Gain * Residue) 
2S When brolcen down into individual equations, the results are as follows: 

'^(x-coordlnate at ^-^Cx^oordlnate at t)+Ga'"[-X(x.cooRilnat© at t-1)+ 

30 X*^ 1 

'^(measured x-coordlnate)i 

^(x-ooordfnat©att)=Xfx.coordinateatt) + ^^'"[-Xfx-coordlnate att-1) + 

X® 1 
'^(measured x-co6rdinate)J 



35 



^(x-coordinateatt)=X(^^rti„ateatt)+ ®a'"[-X^^rtfnate at t-l) + 
^ ^(measured x-coordinate)l 

att)= X,9 a„,+ Gain[-Xj; ^ti.) + xj^^^ g,] 

Xfe at t) = Xfe e. .) + Gain[-Xfe a. + xf^^ 

Xpeait) = '^fea.M, + Galn[-X*=ea„.i> +Xf„^„^^3,j 

m KSaT"™"' " ""^"'"'^ ""'^ ^ ^^^^^^^"^^ "'^ ^-"^ S represents the 

4. Generate Combined iMotion Estimate 
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Covariance Residue Matrix = 
[Measurement Matrix * Prediction Covariance Matrix * 
transpose(Measurement Matrix)] + Measurement Noise 

[0080] Next, the actual likelihood for each motion vector Is calculated. 



(C) 2 2 

Likelihood( H ) = e-<'«^-"*^> 
(S ) 



There is no offset in the preferred embodiment of the invention because it is assumed that offsets cancel each other 
out, and that the system's 16 processes are zero-mean Gaussian signals. Sigma represents variance, is defined in 
the implementation phase of the invention by a human developer. It is l^nown in the art how to assign a useful value 
for Sigma by locking at data. 

20 [0081] Next, mode probabilities are updated from the likelihood generated above and the pre-defined markovian 
mode probabilities discussed above.. 
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pC _ pC-C ^ pS-C ^ pH-C 
pH _ pH-H ^ P&-H ^ pC-H 
pS _ pS-S ^ pH-S ^ pC-S 

The equations for the updated mode probabilities are as follows, where L represents the likelihood of a particular mode 
as calculated above: 

Probability of mode Stationary = 
1/[L^ * (P^ + P"-^ + P^^)+l" • (P"-" + P^" + P^")^.L^ * (P^ + P^-° + P"^ + 

pC-Sjj. .^pS-S ^ pH-S J 

Probability of mode Human = 

1/[L^ • (P^= + p"-« + P^) +l" *(p"^ + P^" + P^"kL° -(P^-^ + P^^ + P"^)I* l" 

.(pH-H ^ pS-H ^ pC>Sj 

Probability of mode Crash = 
1/[L® '(p^® +P""^ + P°^)+l" *(p"""+P^"" + P°-")-aP * (P^^^ +P^ + P""*')]* L*^ 

[0082] The combined motion estimate Is ultimately calculated by using each of the above probabilities, in conjunction 
with the various motion vector estimates: 
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X = Probability of mode Human * x""^ 
+ Probability of mode Crash* X^™^^ 
+ Probability of mode Stationary* x^*^**"*^ 

X is any of the motion variables, including a velocity or acceleration derivation. 
H. ENERGY PREDICTOR 

Kinetic Energy = * Mass * Velocity^ 

mmwmmMm 



Claims 
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5. 



An image processing system as in claim 1. said tracking and predicting system further comprising: 
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a shape tracker and predictor subsystem [48], wherein said shape tracker and predictor subsystem tracks and 
predicts characteristk^ indicating the shape of the occupant; and 

a motion tracker and predictor subsystem [50], wherein said motion tracker and predictor subsystem tracks 
and predicts characteristics Indicating the motion of the occupant. 

5 

6. An image processing system as in claim 5, wherein the characteristics tracked and predicted by said shape tracker 
and predictor subsystem comprise: 

a height [58] of the occupant; and 
10 a shape of the occupant's upper torso. 

7. An image processing system as In claim 5, wherein said characteristics tracked and predicted by said motion 
tracker and predictor subsystem comprise: 

IS a distance [60] between the occupant and the airisag; and 

a forward tilt angle [62]. 

8. An image processing system as in claim 5, wherein said shape tracker and predictor utilizes predefined shape 
states [72. 74, 76]. 

20 

9. An image processing system as in claim 8, wherein said predefined states comprise 

a state of leaning left [80]; 

a state of leaning right [82]; and 

a state of sitting generally centered [78]. 

25 

10. An image processing system as In claim 5, wherein said motion tracker and predictor utilizes predefined motion 
modes [102, 104, 122]. 

11. An Image processing system as in claim 10, wherein said predefined motion modes comprise: 

30 

a mode of crashing [122]; 

a mode of being stationary [1 02]; and 

a mode of being human [104]. 

35 12. An Image classification system as In claim 5, wherein said shape tracker and predictor comprises: 

an update shape predictor [126]; 

an update covariance and gain matrices generator [128]; 
an update shape estimator [1 30]; and 
40 a combined shape estimate generator [1 32]. 

13. An image cfasslficatlon system as in claim 5, wherein said motion tracker and predictor comprises: 

an update motion predictor [134]; 
4S an update covariance and gain matrices generator [136]; 

an update motion estimator [138]; and 
a combined motion estimate generator [140]. 

14. An image processing system as in claim 5,~ wherein said shape tracker and predictor determines a sideways tilt 
so angle [^] of the occupant. 

15. An Image processing system for use with an airtsag deployment system having a seat [20], an occupant [1 8] in the 
seat, an optical sensor [22], an alrtsag [36], a segmentation device [30], an airisag controller [32], said image 
processing system comprising: 

55 

a tracking and predicting system [46], wherein said tracking and predicting system tracks a plurality of variables 
regarding an occupant; 

an ARZ intrusion indicator [52], wherein said ARZ intrusion indicator determines which point of the occupant's 
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can generate estimates 



I more frequent basis than the sensor can capture occupant images [38]. 

19. An image processing system as in claim 1 5. wherein said tracking and predicting system funher comprises: 

pisnr.rsrrsK 

20. An image processing system comprising: 

a sensor [22] for capturing an image of a seat area- 

^^^^ 

21. A method tor predicting the movement of an Image [38], comprising the steps of: 

uSSlnte^^/S"' Tr^'"^' '^"'""^ ^ °' ""^9^ character^tics to incorporate past meas- 

using sa.d updated characteristic predictions to detemiine whether the ImaS w^l enter an Al^!*" 
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Fig. 2 
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